I struggle to see how this API can ever stabilize - what one can do when the heap is near the heap limit is very limited, and in particular if you do too much in the callback to incur another allocation, the program would be in a very unstable state. Our current callbacks only do very limited things within the C++, allowing users to do arbitrary things in JS sounds like an API that invites bugs...I think a safer option would be to only write a profile when it reaches the limit, though even then I am not sure how well the profiler works when the heap is about to reach the limit...

I'm also wary on adding a JS callback to be invoked at new heap limit. This callback also takes a heap profile parameter which is taken on a big heap, sounds like exacerbating the problem it wants to fix.

This API is not intended to make JS safe near OOM. It closes a blind spot in the sampling heap profiler: today, if a process starts startHeapProfile() and reaches a fatal OOM, the accumulated sampling data is lost because stopHeapProfile() is never called so the API provides a best-effort chance to retrieve that data before the process terminates.
The samples are collected continuously, so the near-limit event does not take a full heap snapshot or walk every heap object. However profiler->GetAllocationProfile() still materializes the sampled call tree and may allocate, so capture itself is also best-effort.

Why would the application want to save the profile in JS memory when it's likely not relevent for the application itself that may already crash soon? I think a file-backed API would achieve the same thing without making the users waste the remaining memory with (arguably very noisy) diagnostic data.

@joyeecheung Yes I agree with your proposal! I changed the approach to a file-backed one. please feel free to review it

Codecov Report

❌ Patch coverage is 49.60630% with 64 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.14%. Comparing base (4bec191) to head (481ece9).
⚠️ Report is 75 commits behind head on main.

@@            Coverage Diff             @@
##             main   #64676      +/-   ##
==========================================
+ Coverage   90.13%   90.14%   +0.01%     
==========================================
  Files         741      744       +3     
  Lines      242106   242641     +535     
  Branches    45551    45709     +158     
==========================================
+ Hits       218211   218728     +517     
+ Misses      15430    15411      -19     
- Partials     8465     8502      +37